improvement(copilot): make copilot_messages the sole transcript store, remove JSONB dual-write by waleedlatif1 · Pull Request #4826 · simstudioai/sim

waleedlatif1 · 2026-05-31T04:53:27Z

Summary

Stop reading and writing the legacy copilot_chats.messages JSONB column now that all transcript reads are cut over to the normalized copilot_messages table.
Make appendCopilotChatMessages / replaceCopilotChatMessages the primary store and throw on failure (previously best-effort/swallowed — a swallowed write now means permanent message loss).
Persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear (closes the "marker cleared but message lost" window).
Repoint every remaining direct JSONB reader to copilot_messages: workspace VFS task materialization, chat-cleanup file collection, data-drains export (batched per-page assembly, no N+1), chat fork, and superuser workflow import.
Renamed messages-dual-write.ts → messages-store.ts (no longer "dual"); dropped the now-impossible user-existence dedup check in finalizeAssistantTurn.
Column drop (ALTER TABLE copilot_chats DROP COLUMN messages) + reconcile-script removal are intentionally deferred to a follow-up migration PR so this can bake with a safe rollback.

Type of Change

Improvement / refactor

Testing

126 unit tests pass across lib/copilot/chat, lib/cleanup, app/api/copilot/chat, app/api/mothership.
tsc --noEmit clean (0 errors), biome clean, check:api-validation passes.
DB-verified on staging: transcripts in copilot_messages are byte-identical and correctly ordered; the JSONB column is no longer written.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-05-31T04:53:33Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 31, 2026 7:53am

cursor · 2026-05-31T04:53:36Z

PR Summary

High Risk
Changes core copilot message persistence, deduplication, and transactional boundaries across chat, stop, fork, and import paths; failures now fail the request instead of silently keeping JSONB as fallback.

Overview
This PR completes the cutover so chat transcripts live only in copilot_messages, not the legacy copilot_chats.messages JSONB column. Writes go through messages-store (renamed from dual-write); DB failures now propagate instead of being swallowed, since a failed write would lose messages.

Persistence behavior changes in several places: user and assistant turns are appended via appendCopilotChatMessages / replaceCopilotChatMessages, often inside the same DB transaction as chat-row updates (stream marker, metadata). finalizeAssistantTurn dedupes using the last row in copilot_messages (not the JSONB array) and commits the assistant message with clearing conversationId atomically. update-messages updates chat metadata and replaces the full transcript in one transaction.

Reads and copies that still touched JSONB now load from the normalized table: loadCopilotChatMessages (exported from lifecycle), chat fork, superuser workflow import, data-drains export (batched assembly per page), workspace VFS task materialization, and chat cleanup file attachment scanning. New chats no longer insert an empty messages array on copilot_chats.

Dropping the messages column is deferred to a follow-up migration so this change can roll back safely.

^{Reviewed by Cursor Bugbot for commit c6d7852. Configure here.}

greptile-apps · 2026-05-31T05:02:03Z

Greptile Summary

This PR removes the dual-write pattern for copilot chat transcripts, making copilot_messages the sole store. Legacy copilot_chats.messages JSONB writes are dropped; all reads (VFS task materialization, data-drains export, chat fork, superuser import, cleanup) are cut over to query copilot_messages directly.

messages-store.ts replaces messages-dual-write.ts with appendCopilotChatMessages / replaceCopilotChatMessages that throw on failure instead of swallowing errors.
terminal-state.ts now persists the assistant turn inside finalizeAssistantTurn's transaction, closing the "marker cleared but message lost" window atomically.
copilot-chats.ts (data-drains) switches to a single batched IN (chat_ids) query per page to avoid N+1, and fork/route.ts wraps the new-chat insert and appendCopilotChatMessages together in one transaction.

Confidence Score: 4/5

Safe to merge with one unresolved defect in the background inbox path where message loss can occur silently.

The core interactive-chat path is correctly hardened: the assistant turn is now persisted atomically inside the transaction in terminal-state.ts, and messages-store.ts throws on failure everywhere it's wired up. The gap is in executor.ts's persistChatMessages, the background inbox path: a try/catch swallows any throw from appendCopilotChatMessages, and the db.update(updatedAt) commits before the append runs, so a failure there leaves updatedAt bumped with no messages written. This is the same defect called out in the previous review round and is still unaddressed in this diff.

apps/sim/lib/mothership/inbox/executor.ts — persistChatMessages still swallows append failures and lacks a transaction wrapping the two writes.

Important Files Changed

Filename	Overview
apps/sim/lib/copilot/chat/messages-store.ts	Renamed from messages-dual-write.ts; now throws on failure; append/replace logic is solid with conflict handling and per-chat deduplication.
apps/sim/lib/copilot/chat/terminal-state.ts	Assistant turn now persisted atomically inside the transaction alongside stream-marker clear, closing the previous race window.
apps/sim/lib/mothership/inbox/executor.ts	persistChatMessages still wraps both the db.update and appendCopilotChatMessages in a try/catch that only logs, silently swallowing message loss; the two writes are also not in a transaction, creating a partial-failure window.
apps/sim/lib/data-drains/sources/copilot-chats.ts	Correctly uses a single batched IN query per page to assemble transcripts with no N+1 regression.
apps/sim/app/api/superuser/import-workflow/route.ts	loadCopilotChatMessages is called serially inside the chat loop (N+1 reads); each import is transactional but batch-loading all transcripts upfront would be more efficient.
apps/sim/lib/copilot/vfs/workspace-vfs.ts	Replaced JSONB read with two correlated subqueries per outer chat row (messageCount + messages), each scanning copilot_messages independently rather than once per chat.
apps/sim/lib/cleanup/chat-cleanup.ts	Now reads file attachments from copilot_messages.content instead of the JSONB column; batched correctly with chunkArray.

_{Reviews (3): Last reviewed commit: "improvement(copilot): make copilot_messa..." | Re-trigger Greptile}

waleedlatif1 · 2026-05-31T07:50:29Z

@greptile

waleedlatif1 · 2026-05-31T07:50:33Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6d718cf. Configure here.}

…, remove JSONB dual-write Stop writing/reading the legacy copilot_chats.messages JSONB column now that reads are cut over to copilot_messages. Make appendCopilotChatMessages the primary write (throws on failure instead of swallowing), repoint peripheral readers (workspace VFS, chat cleanup, data drains, fork, superuser import) to copilot_messages, and persist the assistant turn inside finalizeAssistantTurn's transaction so it commits atomically with the stream-marker clear. The column itself is dropped in a follow-up migration after this bakes.

waleedlatif1 · 2026-05-31T08:08:51Z

@greptile

waleedlatif1 · 2026-05-31T08:08:58Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c6d7852. Configure here.}

cursor Bot reviewed May 31, 2026

View reviewed changes

waleedlatif1 force-pushed the waleedlatif1/copilot-drop-dual-write branch from f1a0d4c to 6d718cf Compare May 31, 2026 07:49

vercel Bot temporarily deployed to Preview May 31, 2026 07:49 Inactive

cursor Bot reviewed May 31, 2026

View reviewed changes

waleedlatif1 force-pushed the waleedlatif1/copilot-drop-dual-write branch from 6d718cf to c6d7852 Compare May 31, 2026 07:53

vercel Bot temporarily deployed to Preview May 31, 2026 07:53 Inactive

cursor Bot reviewed May 31, 2026

View reviewed changes

Conversation

waleedlatif1 commented May 31, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

waleedlatif1 commented May 31, 2026

Uh oh!

waleedlatif1 commented May 31, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented May 31, 2026

Uh oh!

waleedlatif1 commented May 31, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented May 31, 2026 •

edited

Loading

cursor Bot commented May 31, 2026 •

edited

Loading

greptile-apps Bot commented May 31, 2026 •

edited

Loading